Apparatus for detecting a pupil, program for the same, and method for detecting a pupil

ABSTRACT

A pupil detecting apparatus detects an edge of a pupil in an image of a face. An outline of an upper eyelid is also detected. The apparatus selects an effective edge extending along the pupil by removing a part of edge extending along the upper eyelid from the edges detected. Then, the apparatus applies the circle Hough transformation to the effective edge in order to obtain a center of the pupil and a radius of the pupil. Thereby, it is possible to detect the pupil accurately even if a part of the pupil is covered.

CROSS REFERENCE TO RELATED APPLICATION

This application is based on Japanese Patent Applications No.2008-226406 filed on Sep. 3, 2008, and No. 2008-226407 filed on Sep. 3,2008, the contents of which are incorporated herein by reference in itsentirety.

FIELD OF THE INVENTION

The present invention relates to a pupil detection, and may be appliedas a form of an apparatus, a program and a method.

BACKGROUND OF THE INVENTION

Conventionally, a technology that calculates a sight direction bydetecting a region of an eye and/or a location of a center of a pupilfrom a face image captured by a camera is known. For example, in aninvention in the patent document 1 listed below, a center of a pupil isdetected by using the following process. First, a region of an eye on animage is detected by a template matching method. An edge or edges aredetected within the region'of the eye detected. Then, the circle Houghtransformation is applied to the detected edge(s) in order to obtain acircle equation and to detect a center of the pupil. Then, the apparatuscalculates a sight direction based on a relative relationship betweenthe region of the eye and the center of the pupil.

Patent document 1: JP-2002-282210-A

However, an outline of a pupil cannot be always detected in a perfectform. A part of a pupil may hide below an eyelid depending on directionof a face or direction of a sight direction. In addition, since an eyeshape also has a personal difference. Therefore, in some cases, a partof a pupil may hide below an eyelid even in a condition where a personfaces front side. For example, there may be a case where an edge isdetected when an upper part of a pupil is hidden below an upper eyelidas shown in FIG. 17A. Here, a pupil is a black region located on acenter of an iris. In the drawings the pupil is illustrated slightlylarge. As shown in FIG. 17B, an edge of the upper eyelid and an edge ofthe pupil are seemingly detected in a continuous form. If the circleHough transformation is performed to such edge, a result is influencedby the edge of the eyelid as shown in FIG. 17C. For this reason,detecting process may detect a location of a pupil with error component.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide anapparatus for detecting a pupil, a method for detecting a pupil and acomputer program for the apparatus capable of detecting a pupil withsufficient accuracy even if a part of the pupil is hidden.

The present invention may be appreciated as an apparatus for detecting apupil, a method for detecting a pupil and a computer program. Thecomputer program is stored in a computer readable medium and causes acomputer to detect a pupil in a face image. One embodiment of thepresent invention is a pupil detecting apparatus including anacquisition means, an eye detection means, an edge detection means, anda pupil detection means. The acquisition means acquires an image of ahuman face which is provided by capturing a face. The eye detectionmeans detects an eye from the image acquired by the acquisition means.The edge detection means detects edge in a region of the eye detected bythe eye detection means. The pupil detection means detects a pupil fromthe edge detected by the edge detection means. Further, the presentinvention is provided with an eye shape detection means and a selectionmeans. The eye shape detection means detects eye shape. The selectionmeans selects only the effective edge extending along a pupil from theedge detected by the edge detection means based on the eye shapedetected by the eye shape detection means. The pupil detection meansdetects a pupil from the effective edge selected by the selection means.

According to the embodiment, since a pupil can be detected only, fromthe edge extending along the pupil selected based on the eye shape, theinfluence by the eye shape can be eliminated and a pupil can be detectedwith sufficient accuracy.

The pupil detection means may be configured to compare a first pupildetection result that indicates the pupil detected based on all of theedge detected by the edge detection means and a second pupil detectionresult that indicates the pupil detected based on only the effectiveedge selected by the selection means, and to select one of the firstpupil detection result and the second pupil detection result based on apredetermined criterion for selection.

According to the embodiment, even if a noise component is in dataindicative of the edge, it is possible to detect a pupil with sufficientaccuracy.

The pupil detection means may be configured to compare the first pupildetection result and the second pupil detection result in heights in theimage, and to select one that is located higher. According to theembodiment, since one of the pupil detection results that is locatedhigher than the other one is selected as a selected result, therefore,it is possible to detect the pupil accurately in many situations.

The pupil detection means may be configured to compare the first pupildetection result and the second pupil detection result in size in theimage, and to select one that is larger. According to the embodiment,since one of the pupil detection results that shows larger pupil isselected as a selected result, therefore, it is possible to detect thepupil accurately in many situations.

The eye shape detection means may be configured to detect an outline ofan eyelid as the eye shape. The selection means may be configured toselect only the effective edge extending along a pupil by removing edgeextending along the outline of the eyelid detected by the eye shapedetection means. According to the embodiment, since only the edgeextending along the pupil is selected by removing edge extending alongthe outline of the eyelid, it is possible to reduce the influence of theoutline of the eyelid and to detect a pupil with sufficient accuracy.

The eye shape detection means may be configured to detect an outline ofan upper eyelid as the outline of the eyelid. The selection means may beconfigured to select only the edge extending along a pupil by removingedge extending along the outline of the upper eyelid detected by the eyeshape detection means. According to the embodiment, it is possible todetect a pupil accurately by removing edge extending along the uppereyelid that hides a pupil in many cases. In addition, it is possible todetect a pupil with less amount of process, since a removing region isrestricted to an upper eyelid region.

The eye shape detection means may be configured to detect the eye shapebased on a face shape model. According to the embodiment, the eye shapecan be detected with sufficient accuracy. In addition, since the eyedetection means may be configured to use the face shape model commonly,and to use similar process. It is possible to provide the apparatus withsimple configuration and processing.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional objects and advantages of the present invention will be morereadily apparent from the following detailed description of preferredembodiments when taken together with the accompanying drawings. Inwhich:

FIG. 1 is a block diagram showing a pupil detecting apparatus in a firstembodiment of the present invention;

FIG. 2 is a flow chart of the pupil detecting apparatus in the firstembodiment of the present invention;

FIGS. 3A, 3B, 3C, and 3D are explanatory drawings for using a face shapemodel;

FIG. 4 is an explanatory drawing for performing a fitting by a faceshape model;

FIG. 5 is a flow chart for detecting a pupil circle;

FIG. 6 is an explanatory drawing for showing feature points on an eye;

FIGS. 7A and 7B are explanatory drawings showing feature selectionprocess;

FIGS. 8A and 8B are explanatory drawings showing pupil circle detectionprocess;

FIG. 9 is a block diagram showing a pupil detecting apparatus in asecond embodiment of the present invention;

FIG. 10 is a flow chart for detecting a pupil circle;

FIG. 11 is a flow chart for determining a location of a pupil;

FIGS. 12A, 12B, and 12C are explanatory drawings for showing a secondpupil detection result;

FIGS. 13A, 13B, and 13C are explanatory drawings for showing a firstpupil detection result;

FIG. 14 is an explanatory drawing showing determined pupil locations;

FIGS. 15A, 15B, and 15C are explanatory drawings for showing a firstpupil detection result;

FIGS. 16A, 16B, and 16C are explanatory drawings for showing a secondpupil detection result; and

FIGS. 17A, 17B, and 17C are explanatory drawings for showing aconventional technique.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention are described indetail by referring drawings.

First Embodiment

FIG. 1 is a block diagram showing a pupil detecting apparatus 100 in afirst embodiment of the present invention.

The pupil detecting apparatus 100 is used by being mounted on a vehicle.The pupil detecting apparatus 100 is provided with a control device 1mainly constructed with a computer, and the camera 2. The control device1 includes a known computer readable storage medium in which a programfor functioning the control device as the pupil detecting apparatus isstored. The pupil detecting apparatus 100 may be appreciated as a systemincluding a pupil location using apparatus 3. As shown in FIG. 1, thecontrol device 1 is provided with a face image acquisition section 10,an eye detection section 11, an edge detection section 12, an eye shapedetection section 13, a feature amount selection section 14, and a pupilcircle detection section 15. The camera 2 is a means for taking animage. The apparatus 3 receives data indicative of a location of a pupilin an eye. The apparatus 3 uses the data for controlling a device on avehicle. The apparatus 3 may be appreciated as a pupil data usingdevice.

The face image acquisition section 10 acquires an image data showing aface image from an image capturing board which captures the face imagetaken by the camera 2 and memorizes the data temporarily.

Here, the camera 2 is arranged in a location which can take an image ofa face of a driver who sits down to a driver's seat in the vehicle froma front side of the driver. In detail, the camera 2 is installed in ameter board in an instrument panel, or its neighborhood. The camera 2 isequipped with an image sensor such as a CCD or CMOS type sensor and alens having a focal distance which enables the camera to take a faceimage clearly.

Two lighting devices (not shown) which illuminates the face of thedriver from both right and left sides with near infrared rays areinstalled in the meter board in the instrument panel near the camera 2.By illuminating an eye of the driver with near infrared rays, the pupilof the driver in the region of the eye can be clearly recognized on animage.

The eye detection section 11 detects a region of an eye from a regionwhere a face is placed within the image. The image is obtained as a formof image data which is acquired by the face image acquisition section10. In this embodiment, a face shape model is used as a method fordetecting a region of an eye. For this purpose, the apparatus may use atechnique described in “I. Matthews and S. Baker, “Active AppearanceModels Revisited,” International Journal of Computer Vision, Vol. 60,No. 2, November, 2004, pp. 135-164.”

The edge detection section 12 detects one or a plurality of edges in theregion of the eye detected by the eye detection section 11. In detail,first, the image is converted in a binary form by a predeterminedthreshold. Then, the well-known Sobel filtering process is applied tothe data. As a result, a boundary or a plurality of boundaries ofregions that are different in a luminance and/or a color tone aredetected as the edge of the pupil.

The eye shape detection section 13 detects a shape of an upper eyelid ofthe eye from the region of the eye detected by the eye detection section11. The shape of the upper eyelid may be referred to as an upper eyelidoutline. The face shape model mentioned above also used in the aboveprocessing.

The feature amount selection section 14 selects only one or a pluralityof effective edges or a part of edge which extend along a pupil byremoving some edges or a part of edge which extend along the uppereyelid outline detected by the eye shape detection section 13 from theedges detected by the edge detection section 12.

The pupil circle detection section 15 outputs a pupil detection result.The pupil detection result is obtained by detecting a pupil based ononly an effective edge which is selected by the feature amount selectionsection 14. In detail, a pupil is detected as a circle in the image. Theeffective edge is selected as an edge which is considered to show anoutline of a pupil alone with comparatively high possibility. The pupildetection result includes a set of center coordinates and a radius ofthe circle which is obtained based on only the effective edge. In theabove processing, the well-known circle Hough transformation is used.

Next, processing performed by the pupil detecting apparatus 100 isexplained in detail.

FIG. 2 is a flow chart showing pupil detection processing which isperformed by the pupil detecting apparatus 100. The pupil detectionprocessing is started by turning on an ignition switch of the vehicles.

In S110, the apparatus gets and acquires an image data which shows aface image of the driver taken by the camera 2. The processing in S110corresponds to processing performed in the face image acquisitionsection 10.

In S120, a region of an eye is detected by performing fitting processingof the image data acquired in S110 using a face shape model. Theprocessing in S120 corresponds to processing performed in the eyedetection section 11.

The face shape model is explained by referring to the drawings. FIG. 3Ashows a base shape s0. FIG. 3B shows a shape vector s1. FIG. 3C shows ashape vector s2. FIG. 3D shows a shape vector s3. In FIGS. 3B, 3C, and3D, the vectors s1, s2, and s3 are drawn on the base shape s0 in orderto clarify the starting point of a vector.

In the face shape model, the base shape s0 is predetermined as shown inFIG. 3A. The base shape s0 is expressed by a plurality of triangularshaped meshes showing a front view of a face. The shape vectors arepredetermined as shown in FIGS. 3B, 3C, and 3D. The shape vectorsinclude n vectors si (i=1, 2, 3 . . . , n). The number n is a naturalnumber. The shape vectors si shows direction of the face from the basevector s0. The face shape model expresses a shape of a face byperforming calculation shown in an expression (1) in which the baseshape s0 and n shape vectors si are used.

$\begin{matrix}{s = {{s\; 0} + {\sum\limits_{i = 1}^{n}{pisi}}}} & (1)\end{matrix}$

In the expression (1), pi (i=1, 2, 3 . . . ) represents superpositioncoefficients. The shape vectors si (i=1, 2, 3 . . . ) are vectors thathave starting points on the apexes of the meshes forming the base shapes0. The apex of the mesh may be called as a mesh apex. The shape vectorssi are defined for all the apexes on the base shape s0. In FIGS. 3B, 3C,and 3D, the shape vectors s1, s2, and s3 are illustrated for only a partof the mesh apexes in order to simplify the drawings.

In the face shape model, the base appearance image A0(x) ispredetermined. The base appearance image A0(x) is an image showing anappearance considered as a basic appearance for a front view of a face.The appearance images Ai(x) are predetermined. The appearance imagesAi(x) include m images Ai(x) (i=1, 2, 3 . . . , m). The number m is anatural number. The appearance images Ai(x) show appearance variationsof face from the base appearance image A0(x). The face shape modelexpresses an appearance of a face by performing calculation shown in anexpression (2) in which the base appearance image A0(x) and m appearanceimages Ai(x) are used.

$\begin{matrix}{{A(x)} = {{A\; 0(x)} + {\sum\limits_{i = 1}^{m}{\lambda \; {{iAi}(x)}}}}} & (2)\end{matrix}$

In the expression (2), x shows the two-dimensional coordinates on thebase shape s0. The λi (Lambda-i) (i=1, 2, 3 . . . ) representssuperposition coefficients.

That is, in S120, the apparatus performs fitting processing which variesthe superposition coefficients pi and the superposition coefficients λi(Lambda-i) so that the image obtained by the expressions (1) and (2)fits to and matched with the face image acquired in S110 as shown inFIG. 4.

In addition, the pupil detecting apparatus 100 stores a plurality ofdifferent sets of base shapes s0, n shape vectors si, basic appearanceimages A0(x), and m appearance images Ai(x) depending on sex, age, andethnic groups. The pupil detecting apparatus 100 is configured to enablea user, e.g., a driver, to select one specific set from the plurality ofsets of s0, si, A0(x), and Ai(x) based on the own sex, age, and ethnicgroup.

Then, in S130, after performing binarization processing within theregion of the eye detected in S120 by using a predetermined thresholdvalue, one or a plurality of edges on the image is detected by using theSobel filtering. The processing in S130 corresponds to processingperformed in the edge detection section 12.

Then, in S140, pupil circle detection processing is performed. In theprocessing, a pupil is detected as a circle based the edge detected inS130, and a result of the detection is outputted as a pupil data to thedevice 3. The pupil data includes the center coordinates and the radiusof the circle showing the pupil. Then, the apparatus completes the pupildetection processing.

Next, the pupil circle detection processing performed in S140 isexplained in more detail. FIG. 5 is a flow chart for detecting a pupilcircle.

In S210, an upper eyelid outline is detected as an eye shape using theface shape model. The upper eyelid outline can be detected based on aplurality of coordinates of the mesh apexes which is related to the eyeon the face shape model. Those apexes are simultaneously located whenthe region of the eye is detected by the face shape model in S120. Thoseapexes may be called as feature points FP of a face.

In detail, as shown in FIG. 6, the upper eyelid outline is detectablebased on the feature points FP1, FP2, FP3, and FP4 which are detectedwhen detecting the region of the eye by the face shape model. Theprocessing in S210 corresponds to processing performed in the eye shapedetection section 13.

Then, in S220, feature amount selection process for selecting onlyeffective edge(s) is performed. The effective edge is considered toextend along a pupil alone. Here, the feature amount selection isperformed by using the feature points FP2 and FP3 indicative of alocation of the upper eyelid acquired in S210. In detail, a straightline L which passes both the feature points FP2 and FP3 is computed.Then, the apparatus defines a straight vertical line which passes thefeature point FP1 corresponding to the location of the outside eyecorner and is vertical to the straight line L. The apparatus locates acrossing point between the straight line L and the vertical line as afirst crossing point. The apparatus defines a straight vertical linewhich passes the feature point FP4 corresponding to the location of theinside eye corner and is vertical to the straight line L. The apparatuslocates a crossing point between the straight line L and the verticalline as a second crossing point. Then, a line segment which has thefirst and second crossing points as both ends is created from thestraight line L. The line segment is shown in FIG. 7A. In FIG. 7A, theedges detected, the feature points FP1, FP2, FP3, and FP4, and a part ofthe straight line L are illustrated. The straight line L in FIG. 7Acorresponds to the line segment.

Next, a region between the line segment and a line located below fromthe line segment by a predetermined amount is defined. The region isalso called an edge removal region. It is possible to assume that theedge(s) which exists in this edge removal region is the upper eyelidedge extending along the upper eyelid with high probability. Then, theupper eyelid edge is removed from all the detected edge(s) in the edgeremoval region. In this embodiment, the predetermined amount fordetermining the edge removal region is 5 pixels. As a result, theremaining edge(s) are selected as the effective edge(s). In FIG. 7B, theedge removal region is illustrated as a square box. Thus, the reason forrestricting within the region which is set by the predetermined pixelsis for removing certainly the edge or a part of edge extending along theupper eyelid. The processing in S220 corresponds to processing performedin the feature amount selection section 14.

Then, in S230, the well-known circle Hough transformation is applied tothe effective edge selected in S220. The transformation determines acircle equation based on the effective edge. In other word, thetransformation detects a pupil as a circle that is estimated based onthe effective edge. The apparatus outputs the center coordinates and theradius of the circle as data indicative of the pupil. FIG. 8A shows theeffective edge selected by removing edge extending along the uppereyelid from the edges in S220. FIG. 8B shows a circle obtained byapplying the circle Hough transformation on the effective edge in S230.The processing in S230 corresponds to processing performed in the pupilcircle detection section 15. Then, the apparatus completes the pupilcircle detection processing.

As mentioned above, according to the pupil detecting apparatus 100 ofthis embodiment, since a pupil is detected by using only the effectiveedge extending along the pupil selected by removing at least a part ofedge extending along the upper eyelid outline, it is possible toeliminate an influence of the upper eyelid outline, and to detect thepupil with sufficient accuracy.

There is almost no possibility that a pupil is covered with things otherthan an upper eyelid. Therefore, the pupil detecting apparatus 100 ofthis embodiment is configured to removes only an edge extending along anoutline of an upper eyelid. For this reason, the accuracy of pupildetection can be efficiently improved with a small processing load.

Since the eye shape detection section 13 detects an eye shape by usingthe face shape model in S210, it is possible to detect the eye shapewith sufficient accuracy. Since the eye detection section 11 alsodetects an eye by using the face shape model in S120, it is possible touse the same processing portions for both sections. It is possible toachieve the apparatus with simple processing.

In the first embodiment, the face image acquisition section 10 providesan acquisition means, the eye detection section 11 provides an eyedetection means, the edge detection section 12 provides an edgedetection means, the eye shape detection section 13 provides an eyeshape detection means, and the feature amount selection section 14provides a selection means. In addition, the pupil circle detectionsection 15 provides the pupil detection means.

Second Embodiment

Hereinafter, a second embodiment of the present invention is describedin detail by referring drawings. In the second embodiment, the samereference numbers as in the first embodiment are used for denotingportions in the second embodiment that are the same as or equivalent tothe portions described in the first embodiment. The precedingdescription may be referenced for the portions denoted by the samereference numbers. In the following description, differences from thefirst embodiment are mainly described.

FIG. 9 is a block diagram showing a pupil detecting apparatus 200 in thesecond embodiment of the present invention.

This pupil detecting apparatus 200 has a pupil circle detection section15 a instead of the pupil circle detection section 15 in the firstembodiment. In addition to the first embodiment, the pupil detectingapparatus 200 is provided with a pupil location determination section16.

The pupil circle detection section 15 a of the second embodiment outputsboth a first pupil detection result and a second pupil detection result.The first pupil detection result is obtained by detecting a pupil basedon all the edge detected by the edge detection section 12. In detail,the pupil is detected as a circle in the image. The circle may bereferred to as a pupil circle. The first pupil detection result includesa set of center coordinates and a radius of the circle which is obtainedbased on all the edge. The second pupil detection result is obtained bydetecting a pupil based on only an effective edge which is selected bythe feature amount selection section 14. The effective edge is selectedas an edge which is considered to show an outline of a pupil alone withcomparatively high possibility. The second pupil detection resultincludes a set of center coordinates and a radius of the circle which isobtained based on only the effective edge. In the above processing, thewell-known circle Hough transformation is used.

The pupil location determination section 16 compares the first pupildetection result and the second pupil detection result, and selects onepupil detection result from the first pupil detection result and thesecond pupil detection result both detected by the pupil circledetection section 15 a. In the selection processing, a predeterminedcriterion is used. The criterion is determined to select most exact oneamong the pupil detection results. In this embodiment, the criterion isdefined to select one pupil detection result that has a center of acircle located on a higher location in the image.

Next, processing performed by the pupil detecting apparatus 200 areexplained in detail.

FIG. 10 is a flow chart showing pupil detection processing which isperformed by the pupil detecting apparatus 200. In the secondembodiment, a step S230 a is executed instead of the step S230 in thefirst embodiment. Further, a step S240 is executed in the secondembodiment.

In S230 a, the circle Hough transformation in applied to all the edge(s)detected in S130. As a result, a pupil is detected as a circle. Thetransformation obtains the center coordinates and the radius of thecircle as the first pupil detection result. The circle Houghtransformation is also applied to only the effective edge selected inS220. A pupil is detected as a circle. The transformation obtains thecenter coordinates and the radius of the circle as the second pupildetection result. The processing in S230 a corresponds to processingperformed in the pupil circle detection section 15 a.

Then, in S240, pupil location determination processing is performed. Inthis processing, one pupil detection result is selected from the firstand second pupil detection result by using the criterion. Again, thecriterion is defined to select the one that has a center located higher.The selected pupil detection result is outputted as a pupil data to thedevice 3. The processing in S240 corresponds to processing performed inthe pupil location determination section 16. Then, the apparatuscompletes the pupil circle detection processing.

Next, the pupil circle detection processing performed in S240 isexplained in more detail. FIG. 11 is a flow chart for determining alocation of a pupil.

In S310, it is determined that whether a height H1 of the centercoordinates of the circle of the first pupil detection result is higherthan a height H2 of the center coordinates of the circle of the secondpupil detection result.

If the height H1 of the center coordinates of the circle of the firstpupil detection result is higher than the height H2 of the centercoordinates of the circle of the second pupil detection result, theroutine branches to YES from S310 and proceeds to S320. If thedetermination in S310 is negative, the routine branches to NO from S310and proceeds to S330.

In S320, since the first pupil detection result has a center that ishigher or equal to the first pupil detection result is outputted as thepupil data to the device 3. Then, the apparatus completes the pupillocation determination processing.

In S330, since it is a case that the second pupil detection result has ahigher center, the second pupil detection result is outputted as thepupil data to the device 3. Then, the apparatus completes the pupillocation determination processing.

The second pupil detection result shows a pupil detected only from theeffective edge selected by referencing a detected data of the eye shape.For this reason, even if the upper part of the pupil is covered andhidden by the upper eyelid, the influence by the eye shape can beeliminated and the pupil can be detected with sufficient accuracy. Thatis, according to the second pupil detection result, the pupil can bedetected with more accuracy in many cases than the first pupil detectionresult which is detected based on all the edge detected by the edgedetection means.

However, if the image has a noise on a pupil, the second pupil detectionresult may not show a pupil with more accuracy than the first pupildetection result.

For example, as shown in FIG. 12A, if both sides of a pupil are hiddenand missed by a noise, such as a reflection light, the edge extendingalong a pupil is detected in a depressed shape at both sides. In thiscase, the apparatus removes an upper part of the edge(s) as the edgesextending along the eye shape as shown in FIG. 12B. Therefore, thesucceeding processing is performed under influences of the depressedshape of the edges and a remaining lower part of the edges. As a result,apparatus detects a pupil as a circle that is located slightly lower andis smaller than the actual pupil as shown in FIG. 12C.

On the other hand, in a case that a pupil is detected based on all theedge(s) detected by the edge detection means, since the upper part ofthe edge(s) extending along a pupil is not removed as shown in FIG. 13B,it is possible to detect a circle that is close to the actual pupil asshown in FIG. 13C.

It is possible to detect a pupil with sufficient accuracy in many casesby comparing the second pupil detection result and the first pupildetection result, and selecting one as a final result based on apredetermined criterion that shows more exact one. For example, even ifa noise component is in data indicative of the edge, it is possible todetect a pupil with sufficient accuracy.

The pupil detection means is configured to compare the first pupildetection result and the second pupil detection result in height in theimage, and to select one that is located higher. Here, comparing theresults in height means that the apparatus compares height locations ofcenters of pupil indicated by the first and second pupil detectionresults. For example, if both the first and second pupil detectionresults indicate circles in the image, the locations of the centers arecompared with respect to the vertical height from the bottom of theimage.

In the case illustrated in FIGS. 12A, 12B, and 12C, in the second pupildetection result, a pupil is detected as a circle that is slightly lowerand smaller than the actual pupil, due to the influence of the lowerpart edge and the inwardly depressed edge. That is, in the processingfor obtaining the second pupil detection result, some part of edge maybe removed even if no part of a pupil is covered with things in order tosuppress the influence of the eye shape. By using such processing, anupper part of edges are usually removed in many cases, since it isconsidered that a shape of a pupil is hardly influenced by things otherthan the upper eyelid. In addition to the above, if a noise is on apupil, the edge becomes a shape inwardly depressed. Therefore, thesecond pupil detection result includes an error component since a pupilis detected as a circle slightly lower and smaller than the actualpupil. This processing is illustrated in FIGS. 12A, 12B, and 12C.

On the other hand, since the first pupil detection result detects apupil from all the edge(s) without removing the upper part, it ispossible to detect a pupil as a circle close to the actual pupil. Thisprocessing is illustrated in FIGS. 13A, 13B, and 13C. That is, as shownin FIG. 14, the first pupil detection result has center coordinates of acircle in a higher location than the other. The first pupil detectionresult shows a circle that is closer to a circle of the actual pupilthan the other. In FIG. 14, C1 shows center coordinates of a pupil shownby the first pupil detection result. C2 shows center coordinates of apupil shown by the second pupil detection result.

In addition, FIGS. 15A, 15B, 15C, 16A, 16B, and 16C show pupil detectionresults in case when a pupil hides behind an upper eyelid without noise.If there is no noise on the image and some part of a pupil hides behindan upper eyelid as shown in FIG. 15A, since all the edge detected by theedge detection section 12 is influenced by the upper eyelid, an edge(s)is detected so that an upper side is located lower than the actual pupilas shown in FIG. 15B. Therefore, if a pupil is detected from the edgeshown in FIG. 15B by using the circle Hough transformation etc., a pupilmay be erroneously detected as a circle slightly lower and smaller thanthe actual pupil in many cases as shown in FIG. 15C.

On the other hand, the second pupil detection result in this case isdetected as a circle near the actual pupil. That is, if there is nonoise on the image and some part of a pupil hides behind an upper eyelidas shown in FIG. 16A, the selection means removes an upper part of thedetected edge and gives a selected effective edge as shown in FIG. 16B.Therefore, the second pupil detection result obtained by applying thecircle Hough transformation to only the effective edge is neverinfluenced by the upper part. Therefore, it is possible to detect acircle near the actual pupil as shown in FIG. 16C.

As mentioned above, in many cases, if a pupil is erroneously detected,the pupil detection result shows lower location of pupil. Therefore, itis possible to detect the pupil accurately in many situations bycomparing the first pupil detection result and the second pupildetection result in heights in the image, and selecting one that islocated higher.

As explained above, according to the pupil detecting apparatus 200 ofthis embodiment, since the apparatus compares the first pupil detectionresult which is detected based on all the edge detected and the secondpupil detection result which is detected based on only the edgeextending along a pupil, and selects one that shows a higher location ofcenter coordinates of a circle of a pupil from the first and secondpupil detection results. Therefore, one of the first and second pupildetection results that shows more exact pupil is selected as a finalresult of detection. It is possible to detect a pupil with sufficientaccuracy while eliminating an influence by a noise on an edge.

In a case where a driver's face is illuminated by near infrared raysfrom both sides, a pupil may have a noise on both sides in many cases.However, a noise is not always appears. In addition, a pupil is hardlycovered with things other than an upper eyelid. Taking those backgroundinto consideration, comparing the first and second pupil detectionresults in height of a center coordinates is considerably effective toimprove accuracy of a pupil detection.

In this embodiment, the pupil circle detection section 15 a and thepupil location determination section 16 provide a pupil detection means.

Other Embodiment

Although some embodiments of the present invention are described above,the present invention is not limited to the above-mentioned embodiments,and as long as it belongs to the technical scope of the presentinvention, it can take various forms.

For example, the pupil detection means may be configured to compare thefirst pupil detection result and the second pupil detection result insize in the image, and to select one that is larger. Comparing theresults in size means that the apparatus compares sizes of pupilindicated by the first and second pupil detection results. For example,if both the first and second pupil detection results indicate circles inthe image, the dimensions of the circles, such as radius and area, arecompared. As mentioned above, an incorrect pupil detection result showssmaller circle than the actual pupil in many cases, such as in a casewhere a noise, such as a reflection light, is appears on both sides of apupil, and in a case where a pupil is affected by an upper eyelid.However, by selecting one of the pupil detection results that showslarger pupil as a final one of selected result, it is possible to detectthe pupil accurately in many situations.

For example, although heights of the center coordinates of circles ofthe first pupil detection result and the second pupil detection resultare compared and the higher one is selected in the above-mentionedembodiment, instead of the above, the apparatus may use a criterion toselect one that has longer radius from the first pupil detection resultand the second pupil detection result. In this configuration, a pupilcan be correctly detected in many situations. The processing in thiscase only requires changing of the criterion used in S310, and otherprocessing are the same as that of the above-mentioned embodiment.

Similarly, in a case that only an area of a pupil is detected in animage, S310 may be modified to select a larger one after comparing areasof the first and second pupil detection results.

In the above-mentioned embodiment, although a pupil is detected as acircle by applying the circle Hough transformation to the edge, a pupilmay be detected as an ellipse detected by applying the least squaremethod.

Although, in the above-mentioned embodiment, the edge removal region isdefined to have 5 pixels, this number of pixels may be obtained based ona height difference between the feature points, such as between FP1 andFP2, or FP3 and FP4.

The edge removal region may be defined based on a curved line, such as aBezier curve and a spline curve. By using a curved line, in a comparisonto a case using a straight line, it is possible to reserve more edgeextending along a pupil, and to reduce a region to be removed.

In the above-mentioned embodiment, although only an edge, extendingalong an upper eyelid outline is removed, it is not limited to this andthe apparatus may be configured to remove an edge extending along alower eyelid. In this case, even if a driver narrows an eye to cover apupil with a lower eyelid, a location of a pupil can be detectedcorrectly.

Further, in the binarization processing in edge detection process,instead of predetermining the threshold value used for binarization, theapparatus may be configured to determine threshold value at each time byusing a well-known discriminant analysis method. In this case, thebinarization processing can be carried out strongly against changing ofillumination, such as cases where a face image becomes bright as awhole, or becomes dark.

In the above-mentioned embodiment, although a driver can choose s0, si,A0(x), and Ai(x) according to own sex, age, or ethnic groups, selectionconditions are not restricted to sex, age, and ethnic groups.Alternatively, s0, si, A0(x), and Ai(x) may be adjusted to be able to beused without being dependent on the conditions of sex, age, ethnicgroups. Thereby, the user of this apparatus can be free from the timeand effort to choose s0, si, A0(x), and Ai(x).

A technique of template matching can be used for detecting a region ofan eye and for detecting an eye shape in stead of the face shape model.

Further, in the above-mentioned embodiments, a single camera is mountedon a vehicle and takes an image of a face of a driver. But, anembodiment of the present invention is not limited to a single cameraconfiguration. A stereo image provided by a plurality of cameras can beused in an embodiment of the present invention.

A purpose and application of the pupil detecting apparatus according tothe present invention is not limited to vehicles, such as a car. Forexample, the pupil detecting apparatus according to the presentinvention may be applied to trains, aircrafts, vessels, medicalapplication machines, simulation machines, game machines, etc.

Although the present invention has been fully described in connectionwith the preferred embodiments thereof with reference to theaccompanying drawings, it is to be noted that various changes andmodifications will become apparent to those skilled in the art. Suchchanges and modifications are to be understood as being within the scopeof the present invention as defined by the appended claims.

1. An apparatus for detecting a pupil comprising: acquisition means foracquiring an image which is provided by capturing a face; eye detectionmeans for detecting an eye from the image acquired by the acquisitionmeans; edge detection means for detecting edge in a region of the eyedetected by the eye detection means; pupil detection means for detectinga pupil from the edge detected by the edge detection means; eye shapedetection means for detecting an eye shape; and selection means forselecting only an effective edge extending along the pupil from the edgedetected by the edge detection means based on the eye shape detected bythe eye shape detection means, wherein the pupil detection means detectsthe pupil based on the effective edge selected by the selection means.2. The apparatus for detecting a pupil in claim 1, wherein the pupildetection means compares a first pupil detection result that indicatesthe pupil detected based on all of the edge detected by the edgedetection means and a second pupil detection result that indicates thepupil detected based on only the effective edge selected by theselection means, and selects one of the first pupil detection result andthe second pupil detection result based on a predetermined criterion forselection.
 3. The apparatus for detecting a pupil in claim 2, whereinthe pupil detection means compares the first pupil detection result andthe second pupil detection result in heights in the image, and selectsone that is located higher.
 4. The apparatus for detecting a pupil inclaim 2, wherein the pupil detection means compares the first pupildetection result and the second pupil detection result in size in theimage, and selects one that is larger.
 5. The apparatus for detecting apupil in claim 1, wherein the eye shape detection means detects theoutline of an eyelid as the eye shape, and the selection means selectsonly the effective edge extending along the pupil by removing edgeextending along an edge of the eyelid detected by the eye shapedetection means.
 6. The apparatus for detecting a pupil in claim 5,wherein the eye shape detection means detects an outline of an uppereyelid as the outline of the eyelid, and the selection means selectsonly the effective edge extending along the pupil by removing edgeextending along the outline of the upper eyelid detected by the eyeshape detection means.
 7. The apparatus for detecting a pupil in claim1, wherein the eye shape detection means detects the eye shape based on,a face shape model.
 8. A computer program which is stored in a computerreadable medium and causes a computer to detect a pupil in a face image,the computer program comprising: acquisition means for acquiring animage which is provided by capturing a face; eye detection means fordetecting an eye from the image acquired by the acquisition means; edgedetection means for detecting edge in a region of the eye detected bythe eye detection means; pupil detection means for detecting a pupilfrom the edge detected by the edge detection means; eye shape detectionmeans for detecting an eye shape; and selection means for selecting onlyan effective edge extending along the pupil from the edge detected bythe edge detection means based on the eye shape detected by the eyeshape detection means, wherein the pupil detection means detects thepupil based on the effective edge selected by the selection means. 9.The computer program claimed in claim 8, wherein the pupil detectionmeans compares a first pupil detection result that indicates the pupildetected based on all of the edge detected by the edge detection meansand a second pupil detection result that indicates the pupil detectedbased on only the effective edge selected by the selection means, andselects one of the first pupil detection result and the second pupildetection result based on a predetermined criterion for selection.
 10. Amethod for detecting a pupil, the method comprising the steps of:acquiring an image which is provided by capturing a face; detecting aneye from the image acquired in the acquiring step; detecting edge in aregion of the eye detected in the eye detecting step; detecting a pupilfrom the edge detected in the edge detecting step; detecting an eyeshape; and selecting only an effective edge extending along the pupilfrom the edge detected in the edge detecting step based on the eye shapedetected in the eye shape detecting step, wherein the pupil detectingstep detects the pupil based on the effective edge selected by theselecting step.
 11. The method for detecting a pupil claimed in claim10, wherein the pupil detecting step compares a first pupil detectionresult that indicates the pupil detected based on all of the edgedetected in the edge detecting step and a second pupil detection resultthat indicates the pupil detected based on only the effective edgeselected in the selecting step, and selects one of the first pupildetection result and the second pupil detection result based on apredetermined criterion for selection.